Derivative Estimates from Simulation of Continuous-Time Markov Chains

Countable-state, continuous-time Markov chains are often analyzed through simulation when simple analytical expressions are unavailable. Simulation is typically used to estimate costs or performance measures associated with the chain and also characteristics like state probabilities and mean passage times. Here we consider the problem of estimating derivatives of these types of quantities with respect to a parameter of the process. In particular, we consider the case where some or all transition rates depend on a parameter. We derive derivative estimates of the infinitesimal perturbation analysis type for Markov chains satisfying a simple condition, and argue that the condition has significant scope. The unbiasedness of these estimates may be surprising-a "naive" estimator would fail in our setting. What makes our estimates work is a special construction of specially structured parameteric families of Markov chains. In addition to proving unbiasedness, we consider a variance reduction technique and make comparisions with derivative estimates based on likelihood ratios.